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ABSTRACT 



Images in an image database are searched in response to 
queries which include the visual characteristics of_ the 
images such as colors, textures, sha pes. ™* gjyes, gg 
by_ jextual tags ^ appe nded to the images. Qu eries are •con- 
structed in an image'query construction area in response to 
values of representations of the visual characteristics and to 
locations of the representations in the image query construc- 
tion area. 
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IMAGE QUERY SYSTEM AND METHOD 

CROSS-REFERENCES TO RELATED 
APPLICATIONS 

This application is a continuation-in-part of U.S. patent 
application Ser. No. 07/973,474, filed Nov. 9, 1992, for 
"IMAGE QUERY SYSTEM AND METHOD", B. Bcitel ct 
al, now abandoned. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates to computer-based system which 
provides access to an image database by use of queries based 
on image characteristic's" ~ - 

2. Discussion of the Related Art 
Relational databases which use queries to request retrieval 

of database information are well known. A query is a 
language-base ^ re quest for information from a database 
based upon spec ific^ondit ions. For example,~a query may 
request a-rrsrof all bank customers whose account balances 
exceed a certain value. 

A recent advance has freed the query function from 
language-based user operations. Tools are available in 
menu-driven dialog interfaces which enable a user to con- 
struct queries by manipulatin g icons in a di alog box on a 
displaj ^creen . The icons are arranged to symbolize database 
search requests. 

To date, database systems which store images have not 30 
provided an efficient means of s&arching_a^atabas e by oth er 
t hanlextual attributes of the imag es. HoweverTthis" requires 
the expenditure of sign ificant resour ces t o annotat e images 
with text. Furthermore, limiting queries to textual armota- 
tibnsTorgoes the possibility of searching an image database 35 
by using pixel information from the actual images whic h are 
.stored i n theH atabase. 

Much work has been expended to enable digital comput- 
ers to rec o g nize images by features which they contain. 
Some success has been achieved in having digital computer s 
identi fy objects in images. H. Tamura, et al. have reported an 
advance in computational detection of image textures in 
their article entitled 'Textural Features Corresponding to 
Visual Perception", IEEE Proceedings, Vol. SMC-8, No. 6, 
June 1978, pp. 460-473. 

Similarly, for characterizing shape, various computational 
methods have been proposed. An example is the algebraic 
moment invariants of G. Taubin and D. B. Cooper Recog- 
nition and Positioning of Rigid Objects Using Algebraic 
Moment Invariants", Geometric Methods in Computer 
Vision, SPIE Vol. 1570, pp. 175-186, 1992. 

Several methods have been reported for querying com- 
puter image databases based ojjjhfi- colois and textu resJLhat 
oc cur in the imag es. The most common basis is color. Two 
cases occur. The first is when colors are selected by a user. 
from a color wheel o r oth er color p icker, and images 
containing the selected colors are retrieved. An example of 
this is given in E. Binaghi, et al, "Indexing and Fuzzy 
Logic-Based Retrieval of Color Images", Visual Database 50 
Systems, II, IFIP Transactions A-7, pages 79-92, Elsevier 
Science Publishers, 1992, in which not only color but also 
percentage coverage of that color and compactness of its 
coverage are used as query conditions. 

The second case is when a color image is specified, and 65 
similar images are to be retrieved. A method for doing this 
is described in M. J, Swain, et al, "Color Indexing", Inter- 



40 



45 



50 



55 



national Journal of Computer Vision, 7(l):ll-32, 1991. The 
method uses ''histogram intersectioiC-in-which a color 
histogram is computepLfor_eacb imagein-the-database, and . 
a corre^SSmg color Jjistogram is coinpiited-forahe-query 
image. These histograms are computed over a quantized 
version of the available color space, giving, for example, 
256 bins in the color histogram, A measure of similarity is 
defined for two histograms, and a query is run by computing 
the similarity between the query image histogram and the 
histogram of each image in the database. 

A more sophisticated method for retrieving images similar 
to a given image is given in Mikihiro Ioka, "A Method of 
Defining the Similarity of Images on the Basis of Color 
Information", Technical Report Kr-0030, IBM Tokyo 
Research Lab, 1989. Here, each image in the database 
(actually, the subimage of each image containing a single, 
dominant object in the image) is partitioned into blocks, for 
example, 25 blocks. Within each block, the reduced bucket 
histogram,h, (say, 256 buckets) is computed. Given a query 
image or object, it is also partitioned into the same number 
of blocks and the histograms computed. A similarity mea- 
sure $fr gu ,ry_ima 8 *>d a t<* a »_u<n} is defined on the color 
histograms computed in the blocks, and the measure is 
extended to images as: 



S(query_jmage ( databasc_image>=X 6todt , ^^ 9TY _f U 

ttrm) 



This method thus takes into account the distribution of 
colors in two images, as well as their spatial position, since 
the colors must not only match, but match at exactly the 
same positions to give a good matching score. 

SUMMARY OF THE INVENTION 

The general problem solved by this invention i sJhat of 
retrieving images from an on-line image database. St andard 
methods use image identifiers and/or text or k eywords 
associated with each image. O ther approaches, such as those 
described in the prior art, use the colors that appear in the 
images. 

Often a user remembers several objects in an image, 
including their approximate color and position, or the user 
may simply request "blue at the top, white at the bottom" in 
an attempt to retrieve a beach scene with sky at the top and 
sand at the bottom. This invention specifically invests a user 
with th e ability to automatically retrieve an image from a 
la rge collection Of ima pes hv approximately specifying the 
co lors or other image characteri stics nf area s that occur in 
the image and the appr oximate positions a t which they . 
oc cur. T he features for oniy a fe w ar eas in the scene need to 
"^esr ^cified. these features need only be specified approxi- 
mately, a nd their posi tion in the scene need only be specifi ed 
approximately. As compared to the Ioka method in the prior 
art, which requires a complete and spatially exact query 
specification, and was designed for image-to-image match- 
ing, the present invention supports a user-composed ijuery ; 
for image_matc Mng, requires characteristics in only "g~few 
areas to be specified , requires the ir position - to he rmlv 
approximately specified so they can Jbe__5ome what miss- 
positioned, and is economical in terms of storage and 
computation. 

The invention is embodied in a query facility which builds 
a visual query by image content An image query area is 
displayed on a display surface. One or more image charac- 
teristic selection areas are displayed which afford a user a 
means to select image characteristics. An image query is 
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constructed by moving selected image characteristic repre- 
sentations from a s electio n area to_the image quer y area, A 
query engine generates an imape query in res ponse to the 
typ es and posit ions of se lected image characteristic rep re- 
sentati ons in th e im age query a rea. Query responses include 5 
database images with features that corres pond to the selected 
imagecharacteristic representations in the image query area. - 

In a first embodiment, an image query window is pre- 
sented, together with one or more image characteristic 
windows. Each image characteristic window represents a 10 
particular predefined image characteristic and includes a set 
of one or more thumbnails (icons) correspo nding to various 
values of thr i mnpf 1 rh^rnrir ri^ti^ rrpiTifjented fry t he char- 
ac teristic wind ow. The thumbnails repr esent impor tant 
aspec t^ ofthf * imagg hp'm^ criii phi such as color, textu re. l5 
sh ape and a rea. Thumbnails representing image character- 
istics may be dragged to the image query window where 
they may be arranged into a desired spatial orientation which 
corresponds to the positioning of image features. The image 
query window therefore may enclose a set of thumbnails 20 
having layout and spatial characteristics selected and 
arranged by a user into an example scene. 

In the first embodiment, a computer display interface for 
constructing an image query used to access images in a 
database based on image content includes a first selection 25 
window for specifying image color and including a plurality 
of color thumbnails t -a„secgnd selection window for speci- 
fying^magt=sriapfis-afld-4iic^ shape 
thumbnails, a third selection window for specifying image 
textures and including a plurality of texture thumbnails, and 30 
a fourth selection window including a plurality of category 
thumbnails w hich may denote keywords, text, or co nditions 
on alpha-numgjiCJl ata aooooiate a^aatn_an image. The inter- 
face fttrfKeTTncludes an image query construction window 
for composing an image query from the thumbnails by 35 
dragging thumbnails from image characteristic windows and 
dropping them at selected positions on the image query 
construction window. 

In a system including a computer display and cursor 
control means for moving symbols on the display, the first 40 
embodiment embraces a method for constructing an image 
query used to access images in a database based on image 
content The method includes the steps of displaying at least 
one selection window on the computer display which 
includes a plurality of sample image characteristic identifiers 45 
(thumbnails), displaying an image query, cjonstruclion^win- 
dow on the display^an^jjicwmg at least, one sample image 
characteristic identifier from the selection container into the 
image query construction window to denote an i ma g e rnn-* 
tent characteristic. 50 

In a second embodiment of the invention, the image 
characteristic selection area includes at leasjji^cjjlflLpalette 
or wheels cursor-based mechanism enables a user to point 
at a*colorin the selection area, to select the color by clicking, 
and to draw or paint with the selected color in the image 
query area. 

It is therefore an object of this invention to enable 
searching an image database by queries based on pixel 
information from the stored images. 60 

Another object of the invention is to provide a query- 
based technique for locating an image in an image database 
by using characteristics of the image in addition to text 
annotation. 

A still further object is to provide, in a query-by-image- 65 
content system, a visual direct-manipulation interface for the 
construction of database queries based on image content. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The objectives set forth above and other distinct advan- 
tages of the invention will be manifest when the following 
detailed description is read in conjunction with the below- 
described drawings, in which FIGS. 3-10B illustrate a first 
embodiment of the invention, FIGS. 11A-14 illustrate a 
second embodiment of the invention, and: 

FIG. 1 is a block diagram of the industrial environment in 
which the invention is practiced; 

FIG. 2 illustrates how the content of an image is decom- 
posed into tabularized data representations; 

FIG. 3 illustrates thumbnail definition tables which are 
used to construct an image query according to the invention; 

FIG. 4 illustrates construction of an image query using an 
image query construction window; 

FIG. 5 illustrates the organization of a display screen 
when the invention operates according to a first embodiment 
in the industrial embodiment illustrated in FIG. 1; 

FIG. 6 illustrates a representative result of an image query 
shown in FIG. 5; 

FIG. 7 illustrates a context dialogue box displayed in the 
operation of the invention; 

FIGS. 8A-8C illustrate various dialogue boxes displayed 
during the operation of the invention. 

FIG. 9 illustrates a results window displayed in response 
to an image query; 

FIGS. 10A and 10B comprise a flow chart that illustrates 
a method of image query construction according to the 
invention; 

FIGS. 11A and 11B illustrate spatial partitioning of an 
image according to a second embodiment of the invention; 

FIG. 12 illustrates the organization of a display screen 
with a block positional picker when the invention operates 
according to the second embodiment in an industrial envi- 
ronment corresponding to that illustrated in FIG. 1; 

FIG. 13 illustrates a second embodiment display screen 
with a freehand positional picker; and 

FIG. 14 illustrates a display screen showing results of the 
query represented in FIG, 12. 



DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 



SYSTEM CONTEXT 

It will he understood and appreciated by those skilled in 
the art that the inventive concepts set forth in the embodi- 
ments described in this application and pertaining to the 
provision of querying an image database by image content 
may be embodied in a variety of system contexts. One such 
system is illustrated by way of example in FIG. 1 wherein 
a data processing system generally indicated by 10 includes 
a proces sing uni t 12, a video display unit 13, and a cursor 
control system including a screen cursor 15 and a mouse 16. 
The mouse is a conventional user-actuatable cursor posi- 
tioner for providing control inputs to the data processing 
system 10. The data processing system 10 further includes a 
data storage mechanism 17 whi ch may include various 
peripheral drives (not shown) and local memory ( also not 
shown) utilized by the data processing system 10 to execute 
programs, control various hardware and joftwar e entities of 
the system, and to store data. ~ 
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The data processing system 10 may be selected from any 
number of conventional processor devices, including, but 
not limited to, processors commonly found in a class of data 
processing apparatus known as workstations. A preferred 
environment of the invention includes a menu-driven user 
interface 20 including a cursor control 21 and a window 
control 22. The user interface 20 operates conventionally to 
provide a menu-driven visual output on the video display 
terminal 13 consisting of a plurality of "windows" or 
"containers" which are used in the practice of the invention- 
One such window 23 is an image query window, which is 
used for constr uction of queries based on image conten t 
Another such window 24 is an image characteristic window 
containing, in a first embodiment, a plurality of icons 
(herei nafter "thumbnails") 26 and_ 27. The user interface 
further provides through the cursor control 21 the ability to 
control the cursor 15 by movement of the mouse 16. The 
cursor 15 is used in the first embodiment to "drag and drop" 
thumbnails from the image characteristics window 24 to the 
image query construction window 23. A thumbnail which 
has been dragged and dropped in the image query construc- 
tion window 23 is indicated by reference numeral 28. 

Once thumbnails have been dragged and dropped in the 
image query window 23, an option is selected denoting a 
RUN QUERY command represented on signal path 30. In 
response, the identification and locations of all thumbnails 
dragged to and dropped in the image query window 23 are 
provided to an image database management system 31 on a 
signal path 33. 

The image database management system 31 includes a 
query-by-image-content (QBIC) engine 32 which receives 
the RUN QUERY command and thumbnail identification 
and location information and, using the information and 
thumbnail definitions stored at 34, constructs a query which 
is used to search image data representations st ored at 35. The 35 
results of the search are used by the QBIC engine 32 to 
construct a result list of images satisfying the query param- 
eters. The identifications of these images are used to obtain 
the identified images from a set of images stored at 36 and 
the images in the result list are provided to the window 4Q 
control 22 for display in a results window, which is not 
shown. 

In a second embodiment, described below, respective 
areas are displayed for presentation of image selection 
characteristic s and construcflbn of an linage que ry- These 45 
areas may t5e"in the same, or separate windows. The cursor 
control in the second embodiment is used to "point and 
click" on a selected image characteristic. By this means, the 
cursor is invested with the image characteristic and may be 
used in the image query construction area to paint or draw 50 
an image query. Once the query has been constructed, it is 
dispatched by the RUN QUERY command. In the second 
embodiment, the engine 32 has access to (and may include) 
an element 37 for computing image query values from a 
spatial grid in the image query construction area. 55 

FIG. 2 illustrates the structure of an image data represen- 
tation which enables the structure to be searched in c onven- 
tio nal query fashion based on ima^e characteristic s. In this 
regard, for example, an imafi ft 40, identified as image n, 
includ es a picture of a church 44 h aving a bell tower topped 60 
with alriangu lar spir e_45. Image n is decomposed into image 
characteristics includi ng color, tex t ure, size, shape and lay- 
out. Arbitrarily, images or image features may be associated 
witfTtext keywords, image IDs, and categorized hy size, 
origin, and date. Hie decomposed elements of the image 40 65 
aredigiTlZeTr~and tabularized in order to construct the data 
representation 42. Relatedly, consider the spire 45. Using 



conventional graphical toolbox techniques, a mask 48 can be 
defin ed on the image 40 to e nclose the spire 45 , the mask 
itself can be characterized by its shape and the pixel data 
covered by the mask can be analyzed to determine its color 
and texture characteristics. In particular, the color of the 
spire 45 can be decomposed to average red, green and blue 
values, and texture parameters (TPARMs) can be obtained 
by known methods, one of which is described, for example, 
in the Tamura et al article discussed in the Background 
section of this application. The size of the mask is simply the 
pixellated area enclosed by the mask 48, while the shape of 
the enclosed spire 45 can be characterized as shape param- 
eters (SPARMs) by known methods, such as those disclosed 
by Taubin and Brown, discussed in the Background. The 
location of the mask may be defined by the offsets in the x 
and y directions from an origin at the upper-left hand corner 
of the image 40 to a point in a bounding box 46 enclosing 
the mask 48. This assumes that the image is displayed as a 
raster-scanned, two-dimensional array of pixels in which the 
pixels are scanned, line-by-line, beginning with the upper- 
left hand pixel. Last, the image, or the individual mask to 
which it is decomposed can be assigned a text/ke yword 
category. For exam ple, if the image n is onel mage of a set 
of images all of which hav e to do with ecclesiastical" 

flrnhit<>r*iiTft ] thft mask cnntAinirigTKe spire might be catego- 
rized b y the single word "spi re". 

In 33s manner, the image 40 is decomposed into the data 
representation 42. The data representation 42 may be orga- 
nized conventionally as a tabularized file for image n whose 
rows correspond to the masks into which the image is 
decomposed and whose columns corresponds to image 
characteristics. 



FIRST EMBODIMENT OF THE INVENTION 

Thumbnail data representations (definitions) which are 
used in the first embodiment of the invention are stored as 
described with reference to FIG. 1, and have the general 
form illustrated in FIG. 3. In FIG. 3, thumbnail definitions 
fo r color, tex tur^ sigft. shape, and catep;onf arf, frHWAH, 
re spectively, by reference number 60, 62, 64, 66 and 68. 
Each thumbnail definition represents one of a plurality of 
thurnbnaiLclefinitions for a particularjTeiiexenced^ 
istjcJPor example, the thumbnail definition 60 is a data 
representation for a color thumbnail which may be denoted 
as adiumbn aiLM re presenting a color M. In this regard, M 
would be theidentity of a thumbnail contained in a color 
characteristic window presented on the display 13 in FIG. 1 
during query construction. The definition for the color 
thumbnail Mis a data structure indexed by the identity of the 
color thumbnail and including average intensity values for 
the red, green and blue component of the color M. The 
definition 62 for texture thumbnail P denoting a smooth 
texture includes the identity (P) for the smooth 10 texture 
thumbnail and a data representation for a smooth texture. 
The data representation can, for example, be represented by 
a set of texture parameters including coarseness, contrast 
and directionality, Similarly, the size thumbnail Q represents 
the size of a pixellated area; the shape thumbnail S can 
include a data representation using an algebraic moment 
invariant; and, the spire category thumbnail includes, for 
example, the text "spire" or additional textural annotation. 
Note that a layout thumbnail is not required since the 
location of the bounding rectangle of any thumbnail in the 
image query window 23 would be available through the 
window control 22 to the query engine 32. 
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It should be evident that at least the characteristics of 
cnl gr, t&xture, size and shape are quantifiable. As discussed 
above with respect to FIG. 2, each image in the database has 
a corresponding data representation in which the calculated 
values jnr these characteristics an - Stftpd fnr ftanh defined 
regionjjfhiterest ( mask) in the i mage. For every mask ) 
specific valuts-foT^the image cnaracteiistics describing 
visual properties for the mask are calculated and entered into 
the data representation. Later, when a query is assembled, an 
obiec t/thumbnail procedur e desc ribed below is e mployed t o 
construc t a descri ption (a "sample image") ot tfle images 
whicti a user wish ed tu ittrieve from the image data base, 
with the query being constructed in terms of values of the 
image characteristics of interest. The query is used to find 
images in the database with image characteristic values that 
are similar to those included in the sample image. To do a 
query, the QBIC engine 32 converts pictorial query infor- 
mation (e.g., the information from the thumbnails and then- 
location) from the image query window into image charac- 
teristic values. 

FIG. 4 illustrates diagrammatically the essential process 
for imag e query construc tion a nd execution where the query 
is mmpnsed hy rtragging'ana'Tlroppin fl thum Bnails! Initially, 
a query is assembled.by draggjng anfl propping character- 
istic m umbn ails in t he image query wi ndow 23. The deposit 
of a mmnbSail creates an object 70. A thumbnail may 
overlap other thumbnails within the same object. For 
example, in the object 70, thumbnails for shape (in 
SPARMS, such as SI,, S2 qi S3 qy . . . ), category (spires), 
color (in color components, such as R 9 , G q , B^), and texture 
(in TPARMS, such as Tl qi T2 q} T3 g ) overlay each other 
within a bounding rectangle which defines the object 70. 
When a query option is selected, the RUN QUERY com- 
mand is sent, together with a description of all objects in the 
query window 23 to the QBIC engine 32. For each object, 
the description includes the identification of th e thumbna ils 
wit hin the object and the object's location measured bythe 
off serbetween its c enter a nd the center of the im age query 
window 23. WhenWRTJrTQUERY command Isgiven, the 
QBIC engine 32 interrogates the window control 22 as to 
objects which are contained in the image query window 23. 
Each object is described in a data structure, with the data 
structure for the object 70 indicated by reference numeral 
72. Each object data structure includes the identification of 45 
the thumbnails which it contains, as well as the offset values 
significant of its location. The QBIC engine 32 then 
assembles a query whose predicate includes a portion for 
each object contained in the query window. Each object 
portion of the query predicate includes, in any order appro- 
priate to thede^igrij iarameter s which co rrespond to t he set 
of imag e cha racteristics. The values for those parameter&^re 
obtained iromTne thum bnail definition s 34. If an object does 
not contain n Thumbnail tor a p articular chara cteristic, the 
paramet ervaltie-fo r that rha rartPTi stic-i^-HnriefirTEri . 

The query assembled by the QBIC engine 32 essentially 
compares the image characteristic values obtained by image 
query window 23 and the thumbnail definitions 34 against 
the values for the corresponding image characteristics con- 
tained in the image data representations 35. Similarity may so 
be measured, for example, by the euclidian distance (d). For 
each image, a composite euclidian distance (cd) is calculated 
between each image object and similarly located masks. 

A comparison is performed using the usual query tech- 
niques with the following elaboration. For each image data 65 
representation, all objects in a query are compared with 
masks in the data representation, and a composite distance 



between the image corresponding to the data representation 
and the sample image is calculated according to: 
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Thus, the composite distance (cjL .. r - n ) be tween imaee n 
n plp image is foe sum of composite distances! 
ween masks of ti^e imalZnnrl the obiects_otthe i 
Tj UeryT The composite distance between a mask fand an 
Object j is the sum of weighted euclidian distances between 
the c alculated*" image ch arac teristics ( color, texture, size , 

Shapejand the value tor the rnrrft<:prm<tir|j; thumbnail rhar- 

attensticsJThe composite distance between mask i and an 
object is given by equation (2), wherein: 



(2) 



In equation (2), the euclidian distance (d) for each calculated 
i mage characteristic is weighted by a value w which is 



assigned by 3efault or by user action described below when \ 
the sample image is constructed in the image query window! 

33: X 

In the first embodiment, the composite distance calcula- 
tion for a mask is performed subject to conditions. First, if 
the object includes a category thumbnail, the calculation is 
performed only for identically categorized masks. Second, 
the object and mask locations may be compared and thresh- 
olded. In this regard, if a mask is located more than a 
predetermined scalar distance from an object, the euclidian 
distance between the object and the mask is not calculated. 

Similarity between the value of an image characteristic in 
an object of a query image and the value of the image 
characteristic in an image mask can be measured by euclid- 
ian distance with features first normalized by variance (a) of 
the characteristic over the entire image database to correct 
scaling inconsistencies, although other similarity measures 
may be employed. For example, when querying on RGB 
color, the distance between mask i and object j is computed 
as: 



(3) 



40 



(red,- red,) 2 



(green, - green/) 2 



(blue,- blue ; ) 2 



Combining textural, size and shape features is done simi- 
larly, by appropriate variation of terms. 

The inventors contemplate that the distance calculations 
for any characteristic value may be thresholded. Thus, 
assume pres pecification of DISTANCE thresholds such as 
jllustratpfl in J3Q4, If the distance calculations for any 
characteristic yield a distance value which exceeds the 
prespecified DISTANCE value, the calculated value is 
replaced by zero. 

As the QBIC engine 32 executes the query, it assembles 
a results list 80 which identifies images and ranks them 
according to their composite distance values. The results list 
80 may include all images or only a prespecified number of 
the closest images. The results list 80 indexes to the images 
stored at 36, and is passed by the QBIC engine 32 to the 
window control 22. The window control 22 retrieves the 
images in the results list from the image storage 36 and 
formats them appropriately for display in a results window 
on the display 13. 

OPERATION OF THE INVENTION 
ACCORDING TO THE FIRST EMBODIMENT 

The image query process executed by the apparatus 
illustrated in FIG. 1 is guided by the display .of windows 
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illustrated in FIG. 5, which show the invention in a first 
embodiment. Initially, an image query screen 90 entitled 
"Example Image" is. displayed. This corresponds to the 
image query window 23 described above with respect to 
FIGS. 1-4. The example image window 90 is where users 
construct a sample image representing the important aspects 
of the images being sought. The example image window 90 
includes a image query (iq) icon 89 on whic h user ma y 
"click^n gjnn an image query . Alternately, a query object 
may be dragged to the image query icon 89 or an appropriate 
command may typed into a command line (not shown). 

In the first embodiment of the invention, four selection 
.windows (containers) are employed: a color, cAptainer^^ a 
' texture cont *vripr °? n Q Hy? ™"*™ner 93. anri a category 
cqntamer 94 . The containers 91, 92 and 93 correspond to 13 
cMScteristics which ar e innate in an image and which can 
be c alculated as described above from im age .content. The 
catego ry characteristic i s an arbitrary textual tag which can 
be appended to a n image~or an ima gfi nmskrEach of the 
selection containers 91-94 contains one or more icons, 
which are referred to, preferably, as "thumbnails". Egch^ 
thum bnail in one of the jmage characteri stic containers 91. 
92 a nd 93 corresponds to a prespecified value for an image 
characteristic. F or example, the KbU" thumbnail 96 in the 
color selection container 91 corresponds to a precalculated 23 
value f or red taking into account all the components of a 
particular display scheme which contjibjiteJoJhe-fjarticiilar 
shade of red given oy the thumbnail 96. In the texture 
selection container 92, the "GRASS" thumbnail 97 corre- 
sponds to a set of precalculated tcxtural values. In the shape 30 
selection container 93, the 'TRIANGLE" thumbnail 98 
corresponds to a quantized representation of a triangle. * 

A sample image ca n be cons irucledJ3 y__draj3sg ing thumb- 
nails t o^the example image window 90 from the^cojor, 
textu re, shape and _ category s election"^ontainers. For 
example, a "BE ARS^Eumbn^iriOO has been dra gged from 
the category container 94 anri dropped \n the eYamplelmage 
wjndfjw.90. A "WATER" thumbnail 106 has been dragged 
from the texture container 92 and dropped in the example 
image window 90. The locations 101 and 107 of the thumb- 
nails 100 and 106 are determined as described above with 
reference to the example image window origin 108. 

The layout of thumbnails in the sampl e image affects the 
resuUsjrfajjuer^Jior example, placing two bear QTumb nails 
next to each other in the example image window indicates a 
preference for adjoining bear objects in the results set of 
images that will be selected to satisfy the query. 

In order to generate a query based on the thumbnails 100 
and 106 dropped in the example image window 90, a RUN 50 
QUERY option is selected. When the option is selected, a 
query is constructed as described above which consists of a 
predica te with an object porti on denoting "S ears" at a ~ ~ 
loc atiolfcor res ponding tn the poinl ^l-and^uiothfij_pbject 
portion denoung _a "wa U ^texture'with a location at 107 . 55 

FIG. 6 illustrates how the thumbnails 100 and 106 on the 
example image window 90 interact with one another to form 
an image query. In FIG. 6, the 1 'bears/water texture" query 
returns pictures containing bears and water. The query 
results are illustrated in a container 110 shown in FIG. 6. The 60 
three bear pictures were returned by the query because in 
each a bear existed slightly off center and there was water in 
the picture off to the right, which corresponds to the layout 
information for the two thumbnails illustrated in the 
example image window 90 of FIG. 5. The xudex^ofjhe 65 
returne d images is^ pjejex a b ly sorted from best tuwo rsT 
match, and the numoerof images return ed can be con trolled 
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by manipulation of the thumbaaiLattributes of weight and 
di stanire described above. 

ThTexample image window 90 preferably is a rectangle 
that has the same aspect ratio as the stored images. Dragging 
and dropping thumbnails from the color qexture, shapejpd 
cate gory selection contai ners onto the example image win- 
do\v~pennits a user to create an example of the kind'uf Image 
which the user wishes to find. 

The placement of the thumbnail is important because the 
spatial relationshi p of the dropped thumbnails is used as 
described above as part of the query crit eria The charac- 
teristiCTdT each dropped thumbnail may be arithmetically or 
logically combined with other thumbnails in the query 
constructed by the QBIC engine 32. As described above, 
thumbnails may be overlapped. As stated previously, when 
thumbnails of different image characteristics overlap, they 
are considered to be part of the same object. For example, if 
the category thumbnail "BEARS" overlaps a "RED" thumb- 
nail, images containing "BEARS" are selected, and in this 
set, the images are ordered according to their similarity to 
the "RED" thumbnail. If the "BEARS" and "RED" thumb- 
nails did not overlap, images including bears and red in their 
respective locations would be returned. 

A context dialogue box illustrated in FIG. 7 is displayed 
when the cursor is placed on a thumbnail and a preselected 
mouse button is clicked. The context dialogue box provides 
two control bars for setting, respectively, distance and 
weight of the thumbnail . Preferably, the se tting of the 
\ distanc e control thresholds the value of the th umbnail char- 
actensac. in dicating how close the value must Eft lb tifiT 
palue pf~T"roaskcharacteristic in "rde f to inninnV. ftp. 

ira cteristic in a query calculatio n. The setting of the 
teignt contro l determines the weig hting factor for this 
^chaiacteristic. Manifestly, the adjustment or thumonail"val- 
ues for multiple thumbnails can establish a relative w eight- 
ing among thumbnail val ues for query construction. For 
example, suppose, in tne "J3BAR"-and'""WOT I&" example 
described above, a user is very inte rested in image color but 
less int erested in the texture of im ages. In this case, the user"" 
wouI5~Tn crease the color attribute, but lighten the weigh t of 
texm 



FIG. 8A illustrates an example an image menu bar 120 
with an EDIT option pull down 122. The edit option pull 
down allows thumbnails previously dropped in the example 
image window to be manipulated. The UNDO option causes 
the last change in the example image window to be removed. 
For example, suppose that the "BEARS" thumbnail was 
dragged and dropped on to the example image window. 
Clicking on UNDO removes the dropped thumbnail. Select- 
ing the REDO option causes the last undone change to be 
restored. Thus, clicking on REDO would restore the bear 
thumbnail in the example image window which was 
"undone" in the previous action. The REVERT option 
discards all changes made since the example image window 
was opened and restores the view of the last version, which 
may be stored on disk. The CLEAR option deletes all 
selected thumbnails in the example image window. The 
CLEAR ALL option deletes all selected objects in the 
example image window; in this regard, it is the equivalent of 
SELECT ALL followed by CLEAR. The SELECT ALL 
option marks all thumbnails in the example image window 
as selected. "ALL" thumbnails include any that may be 
hidden beneath other thumbnails. The DESELECT ALL 
option unmarks all thumbnails previously selected. 

The VIEW option pull down, illustrated in FIG. 8B, 
controls the visibility of the other selection containers used 
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in image query construction. In this regard, once the 
example image window is selected and an image query is 
initiated, the selection windows are conventionally desig- 
nated as "child" process windows. Thus, when any of the 
VIEW options is checked, the associated child process 
selection window appears. If an option is unchecked, the 
associated child process window disappears. 

The COLOR PICKER option displayed in the pull down 
of FIG. 8B enables the user to set an absolute color using a 
COLOR PICKER process window. This option allows a user 
to establish a thumbnail of any selected absolute color. 
Accordingly, a segmented color pie chart is displayed. To 
select the displayed color, the user picks a pie segment by 
moving the cursor to the segment and clicking a mouse 
button. The selected color is displayed in a "Sample Color" 
box. To fine tune a selection, the user clicks and drags a 
cursor anywhere in the pie chart until the desired color is 
displayed. To display the gray scales of a particular hue, the 
desired pie segment is double clicked and the color is 
selected. A following double-click anywhere in the pie chart 
will return the Color Picker process to the polychrome 
mode. Moving RED, GREEN or BLUE sliders in the Color 
Picker window enables a user to mix a color directly. 
Numbers to the left of the RED, GREEN and BLUE sliders 
correspond to actual gun values. Pressing an HIS switch 
switches the three sliders to, respectively, hue, intensity and 
saturation values. Pressing an OK button dismisses the 
Color Picker dialogue box and displays a thumbnail of the 
selected color in the center of the example image window. 
This thumbnail may then be manipulated like any other. 

The OPTIONS pull down from the menu illustrated in 
FIG. 8Ais shown in FIG. 8C. In this menu, the Layout check 
box enables or disables the spatial relation. When enabled, 
the thumbnail positional relationships to one another are 
accounted for as discussed above in the calculation of mask 
composite distances. If the Layout check box is disabled, the 
composite distance calculations are not thresholded by the 
scalar distances between objects and masks. Effectively, if 
Layout is disabled and the example image window, for 
example, contains two "bird" category thumbnails, then the 
query would select all images in the image database that 
contains at least two birds. The Run Query option corre- 
sponds to the RUN QUERY command that, when selected, 
results in execution of the query. 

Query results are displayed as illustrated in FIG. 9. In 
FIG. 9, results are included in a Query Result window 200 
which includes at least three columns. The first column 
(LT_THUMB) includes a thumbnail for a complete image. 
The second column (LT_MASK) shows a selected sub-area 
of the image with one or more mask outlines. Th e third 
column 202 illustrate^g raphically th* pn^nfisitc distance of 
an imaffTTrom th e input query. The query results are 
displayed in the orueF of most similar to least similar. 

FIGS. 10A and 10B comprise a flow chart describing a 
process for practicing the first embodiment of the invention. 
In FIG. 10A, the process is initiated in step 210 by display- 
ing an example image window such as the window 90 
illustrated in FIG. 5. A query-by-image content procedure is 
initiated in step 211 by, for example, selecting the query 
button 89 in the example image window 90, which issues a 
RUN QUERY command. In step 212, selection windows are 
selected and displayed using the view pulldown illustrated in 
FIG. 8B. Thumbnails are then dragged from selection win- 
dows and dropped in the example image window in step 
213. For as long the RUN QUERY command is not selected, 
the negative exit is taken from decision 214 and the sample 
image being constructed in the example image window can 



be e dited-D J-added-to-m-Uep 213r4V-hen-the RUN QUERY 
command is selected, the positive exit is taken from decision 
214 and, for each image in the database, the procedure 
executes a sequence of steps beginning with decision 216 
5 and ending with step 225. Recall that selection of the RUN 
QUERY command sends a set of data structures describing 
objects in the sample image to the QBIC function. Each 
object data structure is inspected to determine whether the 
obj ect includes a category thumbnail. If so, the positive exit 
10 is Taken from decision 216 and the masks in the current 
image are filtered by the category code in the data structure 
in step 217. Trie decision 216 then establishes the set of 
masks of the current image which will be analyzed for 
composite distance with respect to the image characteristic 
15 values returned in the data structure for this object. The set 
of masks established in the decision step 216 are then 
subjected to the composite distance calculations as described 
above with reference to equations (1H3). In step 218, the 
layout option is inspected. If enabled, the composite distance 
20 calculations for each mask are thresholded by mask/object 
distance in step 220. In step 222, taking account of the 
outcome of decision 218, composite distances between 
objects of the sample image and the filtered and thresholded 
masks of the current image are calculated using the weigh t, 
d istance, and thumbnail values return for the object in step 
l\i Nftict, in stftp 224, the composit e distance for the image 
is c alculated . If the image is not the last in the set of images, 
the negative exit is taken from step 225 and the composite 
distance for the next image is calculated as just described. 
30 Otherwise, the positive exit is taken in step 225, the images 
are ranked according to the results of the composite distance 
calculations in step 227 and the result of the ranking are 
displayed in step 228. 
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OPERATION OF THE INVENTION 
ACCORDING TO A SECOND-EMBODIMENT 



The second embodiment of the invention builds an image 
query by a user-specified "painting" in an image query area. 
40 This embodiment finds images according to a user-specified 
pattern consisting of image characteristics at specified posi- 
tions; it includes components corresponding to elements Of 
the system shown in FIG. 1, as follows:: 

1. Image Storage. Each image is stored as an IxJ array of 
pi xels. Each pixel inclu des one or more multi-bit words 
denoting i mage co lor at th e pixel lncatkm-Leilhcr 
directly or using a color look-up table. 

2. Spatial Partitioning. The pixel array of each image in 
50 the image store is spatially partitioned, such as into a 

rectangular grid, a radial grid providing finer granular- 
ity in the center of the image, or any other quantization 
that fits the image data and application. This partition- 
ing need not be fixed for all database entries. In fact, 

55 dynamic mapping may be used for greatly differing 
resolutions of images, image content based segmenta- 
tion, or user selection at database population time. Two 
examples of a spatial partitioning of the images are 
shown in FIGS. UA and UB. The partitioning of FIG. 

60 11 A is by a regular grid, dividing each image into a 3x4 
array of grid areas. The partitioning of FIG. 11B results 
in finer grid areas in the center of each image. 

3. Feature Computation to populate the database of image 
representations for use during Image Query. For each 

65 image in an image database, using the areas defined by 
the partition in Step 2, a set of image characteristics is 
computed for each area. Possible image characteristics 
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include measures such as imagecolor, text ure, and/or 
edgeconiomv The area and characteristic information 
arestored in the storage reserved for image data 
representations. 
The feature computation algorithm can be represented as: 5 



14 

array of color choices. Any color may be picked from 
the color choice array and placed in any grid position 
of the query composition window. The algorithm 
supporting the picker is: 



For each image I(j) in the image collection 

For each area A(k) defined by the spatial partition 

For each image feature F(l) (such as color features, 
texture parameters, etc.) 
Compute feature F(l) over area A(k) 

End 

End 

Store in the database the set of F{1) for each A(k) 



The result of this step is a table or database such as is 
shown in Tfcble I containing the computed information. 
This example is for r total features, s different areas in 
the spatial partition, and t total images in the image 
collection. The inventors have used F(j.k,l) to represent 20 
the 1-th feature in the k-th area in the j-th image. 



Repeat until the user clicks on the OK button 

If the user clicks on a color box in the left side of the menu 

Make this the current color 
If the user clicks on a grid box in the right side of the menu 
Place the current color in the grid box 

End 

Return an array of the set of grid boxes with their 
associated color 



b. Display a freehand "positional picker". An example 
is given in FIG. 13 in which the user is allowed full 
freehand drawing in the image query window. The 
example shows green (grass) in the bottom portion of 
the image, blue (sky) in the top portion, and a yellow 
circle (the sun). The algorithm supporting this picker 
is: 



TABLE I 



1(1) 


A(1)F(1,U,) F(l,l,2) F(l,l,3). 


...F(J,l,r) 




A(2)F(1,2 ( 1,) F(1,2,2)F(1A3). 


■ • • F(U,r) 




A(3) F(1,3,1,)F<1,3,2) F(l,3,3) , 


..-F(U,r) 




A(4)F(1,4,U F(l,4,2) F(l,4,3). 


. . . F(l,4,r) 




A(s) F(U1) F(l,s,2) FO.s.3) - . 


• F(Ur) 


1(2) 


A(l) F(1.U.)F(U,2) FU.1,3) 


. - - F(l t l,r) 




A(2)F(1,2,UF(1,2 ( 2)F(1A3) 


. . . F(l,2,r) 




A(3) FU^UFU^FU^) 


...FO,3,r) 




A(4)F(1A1.)F(1A2)F(1,4,3) 


. . . FQ,4,r) 




A(s) F(2,s.l) F(2,s ( 2) F(2 t s f 3) . . 


. F(2,s j) 


1(3) 


A(1)F(3,1,1)F(3,1,2)F(3,1,3). 


.-F(3,U) 




A(l) F(t,l,l.) FG.1,2) FG.1,3) . 


- . WU) 




A(2) F(t,2,l.) F(t,2,2) F(t,2,3) . 


. . F(f,2,r) 




A(3)F(t,3,l,) F(t,3,2) F(t,3,3) . 


■ . F(f,3,r) 




A(4) F(t,4,l,) F(t,4,2) F(u4.3) . , 


. - F(f,4j) 




A(3) F(ul) F(t^2) F(t,s,3) ■ - . 


F(t, St r) 
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4. Query specification. To specify a query, obtain an 
approximate specification of the image characteristics 
that occur in some area or areas of the image. Speci- 50 
fication of image characteristics in only a few areas is 
explicitly supported. The specification can be obtained 
from a set of thumbnails, an interactive feature "picker" 
in an image characteristics window, or from another 
image. Single or multiple features may be specified per 55 
area. Examples of feature "pickers" used to obtain a 
query specification are: 

a. Display a block "positional picker* 1 . An example is 
shown in FIG. 12. The example allows a user to 
specify a query based on positional color feature by 60 
explicity drawing or painting them in a block-like 
manner. In this regard, the image query composition 
area 120 is two-dimensionally partitioned in a simi- 
lar, but not necessarily the same manner as the 
images. For example, the grid in the query area 120 65 
is partitioned into 64 grid positions. The color char- 
acteristic area 121 displays a multi-dimensional 



Repeat until the user clicks on the OK button 

If the user clicks on a color box in the left side of the menu 

Make this the current color 
If the user draws in the drawing area in the right side 

of the menu 

Paint the user drawing with the current color 

End 

Return a drawing area containing the drawn, colored shapes 



c. Display an image (call this the query image) and 
let the user outline areas of the image that are 
"relevant". That is, the user draws polygonal "areas 
of interest", and all remaining area is considered 
"don't care". The algorithm supporting this picker is: 



Display an image 

Let the user draw (multiple) polygonal areas on the image 
Return a copy of the image in which pixels in the polygonal areas 
are unchanged, all others flagged as "don't care" 



These examples use color as the positional feature. 
Other features, such as positional texture, may also 
be used. 

5. Similarity matching. For each image in the collection, 
compute its similarity score: 

(a) For each area specified in the query, compute a 
positional feature score that compares the area's 
similarity to the image areas computed in Step 3. 
This score combines both features along with posi- 
tional similarity so that areas with similar features 
get higher scores, dissimilar features get lower 
scores, and areas positionally close get higher scores, 
and areas positionally far get lower scores. The result 
is a score, for each query area, of its positional 
feature similarity within this image. The highest 
scores will be obtained by areas both positionally 
close and with similar features. Indexing techniques 
could be used to increase the performance of search- 
ing for the "best" matches. 

(b) Combine the scores of all query areas to give a 
global score for the image. 

(c) Rank the images by their global scores and return, 
as the results of a query, the images with the best 
scores. 
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An algorithm supporting similarity matching is: 



For each image I(j) in the image collection 

Initialize the similarity score S(j) for this image 

If necessary, map the features of the picker and the features from 

the query (e.g., the picker) to a common spatial grid 
For each are A 00 defined by the spatial partition 

For each image feature F(l)(such as color features, 
texture parameters, etc.) 

If the user specified a value for this feature in this area 
(that is, if Q(l) is not "don't care") 
Compute a similarity score between the picker feature 
Q(l) and the image feature F(j,k,l) for this area. 
The scoring is flexible in that it accounts for 
approximately matching in both the feature 
values and in their position by looking in 
neighboring areas. 
Accumulate this score in S(j) 
Endif 
End 
Endif 

Order the set of similarity scores S(j) 

Display to the user the images with the best similarity scores 



10 



The step above of mapping the picker features and 
query features is necessary if they are given on a 
different spatial grid. For example, if the set of areas 
A(l), . . . , A(s) in Step 2 over which the image features 
have been computed correspond to an 8x8 grid, and the 
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A specific example of the second embodiment is as 
follows: 

IMAGE STORAGE 

As described above, each image consists of an array of 
pixels describing the color of the corresponding image 
location. 

Each image is partitioned into an N X M set of blocks. 
Other partitions may be used, such as irregular partitions, 
partitions with finer resolution in the image center where 
more detail is expected, etc. In this embodiment, blocks 
correspond to the set of boxes used in the image query 
composition window 120 of the user interface. The blocks 
are of approximate size (number of image lines^N) lines by 
(number of image pixels per line/M) pixels. Let B be the 
number of partitioned blocks, M X N. 

FEATURE COMPUTATION 

In this example, color is used as the image characteristic. 
It can be represented as average color or, alternately, as a list 
of the most frequently occurring colors in each block. For 
the latter case, within each block, a k-level color histogram 
(typical values of k are 32, 64, and 256) is computed, and the 
top t values are kept, where typically t<k, giving a table for 
each image of the form: 



TABLE II 



BLOCK 


COLOR 


COUNT 


COLOR 


COUNT . 


. . COLOR 


COUNT 


1 


c,U) 


Pid) 


C!(2) 


P:(2) 


• c,(l) 


Pi(t) 


2 




Pad) 


C2(2) 


P 2 (2) 


• 


P2W 


3 


c 3 0) 


Pad) 


ca(2) 


P3<2) 


. C3O) 


P 3 (0 


4 


c 4 (D 


P4(D 


c 4 (2) 


P4<2) 


■ c 4 0) 


p4« 


B 


Ct(0 


PtO) 


cr(2) 


Pt<2) ! 


. o^t) 


PT<t) 



c,(l) is the color (or index of the color) in the color histogram of the bucket with the largest 
percentage of pixels in the first (ix., upper left) block of the image, p](l) is the percentage of 
the pixels in that image block that fall in this bucket, and, in general, Cj(k) and p,(k) are the 
color and percentage, respectively, for the histogram bucket with the k-th largest percentage of 
pixels in block i. 



freehand positional picker were used, then a 64 element 
query array Q, with one element for each of the 8x8 
grids, can be computed as: 



Divide the freehand drawing into an 8x8 grid 

For each grid area in the freehand drawing 

Find the number of pixels in the area of each possible color 
Find the color C with the largest number N(C) of drawn 
pixels 

If N(C) is larger than some threshold (say of the size 
of the grid area) 

Set the query array element Q(s) for this grid area to C 
Else 55 
Set the query array element Q(s) for this grid 
area to "don't care" 

End 

Similarly, if the image masking picker were used, feature values 
from the query image within the areas covered by the 

drawn polygons are averaged in each area 60 
corresponding to the 8x8 grid. 



In summary, the matching algorithm described above 
consists of two parts. The first is done once per image, for 
example, when an image is loaded into the database, and 65 
consists of computing and storing data about the image. The 
second is the actual matching, done at query time. 



This table (excluding the first column, which is not stored) 
is computed and stored for each image. Its size is B X 2t 
when t«k but most B X k. (When Me, efficient coding 
techniques can be used, such as listing excluded values of 
color, or, when t=k, omitting the values altogether.) 

QUERY SPECIFICATION 

A sample user interface (the one implemented by the 
inventors) for the case of color features consists of the image 
query composition window 120 or blackboard" as shown in 
FIG. 12, divided into an N X M array of boxes. A user fills 
in rough color-and-position estimates by selecting a color 
from the available set of colors in the color characteristic 
window 121 and dropping it in any set of boxes. A color may 
be dropped in multiple boxes, and additional colors can be 
dropped in other boxes. Boxes in which no color is dropped 
are "don't care". An example completed query specification 
is shown in FIG. 12. 

SIMILARITY MATCHING 

Once a set of boxes has been colored, the query is 
initiated. The method finds images in the database that have 
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approximately the specified color in approximately the 
specified position. 

The matching process takes the specifications of a query 
and ranks images in the database according to how well they 
match the query. Inputs to the matching are the query 5 
specifications from the image query construction window, 
and the tabular data described above. Output is a ranked list 
of the images in the database, ranked in order of their match 
to the query specification. 

The matching process requires a color similarity function 10 
(s c ) giving the similarity between any two colors. To define 
this, the process starts with a color distance function. Many 
choices are possible. Euclidean distance in L*a*b* space 
may be used, for example. This distance function is referred 
to as d^color^colorj. From this, the color similarity rune- 15 
tion is defined as: 



s tf (c olor t .color^ 1 -d cl<wiw i t color2)/(maxd c ) 



(4) 



where max d c is the maximum value of d c (color lf color 2 ) 
over all color pairs. Similar colors have d c near zero and s c 20 
near 1; different colors have d c large, and s c near 0. 

The query specification from the image query construc- 
tion window is a list q„ i=l, . . . , MxN giving the selected 
color for each block, q, can be specified as a triple of red, 
green, and blue values (r,g,b), an index into a pre-selected 25 
color table, or any convenient method of specifying a color. 
Some method is used to indicated a "not specified" or "don't 
care", such as a special value of q, or a separate binary array. 

For each image in the database, matching is performed as 
follows. For each block for which q, is not "don't care", the 3Q 
similarity between the color q ( in the i-th block and the 
colors in the corresponding block of image h is computed. 
An example block similarity function is: 



h=l 



(5) 
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To make the similarity matching robust to position, the 
process also searches in neighboring image blocks for good 
color similarity, and extends the similarity matching to 
match query block i with image block j: 



40 



4j= ^PjHfydqitC/W) 



(6) 



Using this position-robust measure, the similarity score 
s* for block i in image H is: 



45 



_ A _ max h 

Si ~J c ncighborhood_of_r 



(7) 



where w tf is a weighing function that increases the similarity 
of blocks close to block i. In this implementation, a neigh- 
borhood is used consisting of i and its 8-neighbor blocks, 50 
and w,y=V2 for i=j, w^/3 for 8-neighboring j values, while 
w^ for the other blocks. 

The final position color similarity for image h over all q ( is: 



(8) 



55 



where the product is over all i for which q. is not "don't 
care". 

Query results are shown in FIG, 14. 

In the view of the above description, it is likely that 
modifications and improvements will occur to those skilled 60 
in the art which are within the scope of the accompanying 
claims. 

We claim: 

1, A computerized method of retrieving images from an 
image data base, comprising the steps of: $5 
receiving a depictive specification of one or more image 
characteristics of a search image, the specification 
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including a type, a location, and a weight of the one or 

more image characteristics; 
executing a query for images in the image data base that 

are similar to the search image by: 

determining whether an image in the image data base 
has an image characteristic of the type and near the 
location of the one or more image characteristics of 
the search image; and 

determining similarity between the image and the 
search image according to the weight of the one or 
more image characteristics of the search image; 
accumulating a set of images determined to be similar to 

the search image; and 
providing the set of images ranked in an order according 

to a degree of similarity to the search image. 

2. The method of claim 1, wherein the one or more image 
characteristics include color. 

3. The method of claim 2, wherein the one or more image 
characteristics include texture. 

4. The method of claim 2, wherein the step of receiving 
the specification includes: 

displaying an image query window and receiving a rep- 
resentation of the one or more image characteristics in 
the image query window; 

the location of the representation into the window defin- 
ing the location of the one or more image characteris- 
tics. 

5. The method of claim 4, wherein the representation is an 
icon. 

6. The method of claim 2, wherein the step of receiving 
the specification includes partitioning the search image into 
a plurality of partitions, each partition being a specification 
of the one or more image characteristics according to 
location. 

7. The method of claim 6, wherein the step of partitioning 
includes outlining areas of the search image, each outlined 
area being a specification of the one or more image char- 
acteristics according to location of the outlined area. 

8. The method of claim 7, wherein the areas are polygonal 
areas. 

9. The method of claim 2, wherein the type of the one or 
more image characteristics is a color. 

10. The method of claim 9, wherein the color is defined 
by a group of color components. 

11. The method of claim 10, wherein the group of color 
components is a group of (RGB) components. 

12. The method of claim 1, wherein the step of receiving 
the specification includes receiving specifications of a plu- 
rality of image characteristics and the step of determining 
similarity includes, for each specified image characteristic, 
calculating a composite distance between the specified 
image characteristic and an image characteristic located near 
a location specified for the specified image characteristic, 
and combining composite distances for the plurality of 
specified image characteristics according to weights speci- 
fied in the specifying step. 

13. The method of claim 12, wherein the plurality of 
specified image characteristics includes color. 

14. The method of claim 13, wherein the plurality of 
specified image characteristics includes texture. 

15. The method of claim 1, wherein the one or more image 
characteristics include color and the step of determining 
similarity includes, for the type of color (colo^), determi- 
nation of color similarity according to a color similarity 
function s c (colar^ colorj, where color 2 is any color and so 
is near a value of 1 when colo^ is similar to color 2 , and s c 
is near a value of 0 when color! is different than color 2 . 
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16. The method of claim 15, wherein the step of deter- 
mining similarity includes determination of color similarity 
according to: 



Jc(colori, colori) = 1 - 



rfctcolort, color?) 
maxde 
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where d c is a color distance function in which d f is near zero 
and s c is near 1 when coloTj and color 2 are similar, while d c 
is large and s c is near zero when colo^ and color 2 are 
dissimilar. 

17. The method of claim 1, wherein the weight is the 
weight of the one or more image characteristics with respect 
to other image characteristics. 

18. The method of claim 17, wherein the step of deter- 
mining similarity includes determining a composite distance 
between the image and the search image by combining a 
plurality of image characteristic distances according to a 
plurality of weights. 

19. A computerized method of managing an image data- 
base, comprising the steps of: 

constructing an image database by: 
storing a plurality of images in an image store; and 
storing a plurality of image data representations of the 
plurality of images; 

receiving a specification of one or more image character- 
istics in a search image, specification of the one or more 
image characteristics including a type and a location; 

querying the image database for images that are similar to 
the search image by: 

identifying from the image data representations one or 30 
more images that include the type specified in the 
search image near the location specified in the search 
image; and 

determining similarity between each image of the one 
or more images and the search image; 35 
accumulating a set of images determined to be similar to 

the search image; 
ranking the images of the set of images according to a 

degree of similarity to the search image; and 
providing the set of images, in ranked order, from the 40 

image store. 

20. The method of claim 19 wherein the step of receiving 
the one or more image characteristic includes receiving a 
plurality of specifications for a plurality of image charac- 
teristics in the search image, each specification including a 45 
weight of an image characteristic, and the step of determin- 
ing similarity includes, for each image characteristic, cal- 
culating a composite distance between the image character- 
istic and a corresponding image characteristic located in an 
image of the one or more images near a location correspond- 
ing to a location specified for the image characteristic, and, 
for the specified image characteristics, combining composite 
distances according to weights specified in the search image. 

21. The method of claim 20, wherein each image data 
representation represents a single image. 

22. The method of claim 19, wherein the step of receiving 
a specification includes: 

displaying an image query window and moving a repre- 
sentation of the one or more image characteristics into 
the image query window; 



50 



55 



the location of a representation in the window being a 
specified location. 

23. The method of claim 22, wherein the representation is 
an icon. 

24. The method of claim 19, wherein the step of receiving 
a specification includes outlining areas of the search image, 
each oudined area specifying the one or more image char- 
acteristics according to location. 

25. The method of claim 24, wherein the areas are 
polygonal areas. 

26. The method of claim 19, wherein the step of storing 
a plurality of image data representations includes, for at least 
one image in the image data base: 

partitioning the image into a plurality of partitions; and 
storing a data structure for each partition representing a 
respective image characteristic contained in the parti- 
tion. 

27. The method of claim 26, wherein the step of receiving 
a specification includes partitioning the search image into 
partitions and specifying one or more image characteristics 
in one or more partitions. 

28. The method of claim 19, wherein the one or more 
image characteristics include color and the step of deter- 
mining similarity includes, for the type of color (color] ), 
determination of color similarity according to a color simi- 
larity function s c (color^ color^, where color 2 is any color 
and s c is near a value of 1 when color! is similar to color 2 , 
and s c is near a value of 0 when color 2 is different than 
color 2 . 

29. The method of claim 28, wherein the step of deter- 
mining similarity includes determination of color similarity 
according to: 



j c (colort, coloi^) = 1 - 



t/ c (colori,color2) 
maxdc 



where d c is a color distance function in which d c is near zero 
oral s c is near 1 when colorj and color 2 are similar, while d c 
is large and s c is near zero when colorj and color 2 are 
dissimilar. 

30. The method of claim 19, wherein each weight is the 
weight of a respective image characteristic with respect to 
other image characteristics. 

31. The method of claim 19, wherein the one or more 
image characteristics include color. 

32. The method of claim 31, wherein the type of the one 
or more image characteristics is a color. 

33. The method of claim 32, wherein the color is defined 
by a group of color components. 

34. The method of claim 33, wherein the group of color 
components is a group of (RGB) components. 

35. The method of claim 31, wherein the one or more 
image characteristics include texture. 
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